Templeton's Known Limitations

Most software documentation only tells what the program can do, but we feel that it is equally important to know what it cannot do.

Due to the vast options available on most web servers, Templeton is not capable of retrieving some types of documents. These known limitations effect virtually all types of web robots and are described below. Many of these limitations are expected to be corrected in later releases, although currently no dates have been announced.

Protocol Limitations

The following limitations are due to the HTTP protocol and its implementation.

Templeton attempts to identify file types using HEAD requests. Some servers (especially database systems) return errors for HEAD requests but complete documents for GET commands. Templeton may incorrectly determine a file as unretrievable if the HEAD request fails.
Some servers do not return error 404 (or 4xx) for invalid documents. A few servers have been seen returning HTML documents containing a text error message and return code 200 (OK). Templeton will incorrectly process these documents; the text error message with return code 200 will be written to a file and any links within the document will be identified and processed. (In the case of imagemaps, these documents will not be processed.)
Templeton does not support POST commands.
Netscape "client pull" (aka refresh) tags are not supported. These tags are usually in the form: <META HTTP-EQUIV="Refresh" CONTENT="?;URL=URL">
Templeton does not support text entry fields, selections, or pushbuttons.
Image Maps are "clickable" pictures. By clicking on a portion of the picture, you activate a link to another HTML document. There are currently four ways to resolve imagemap URL addresses:
1. Client-Side processing requires the URL to be identified by the client application. Templeton supports this method.
2. CERN/NCSA server-side mappings use the URL path to describe the mapping program (generally imagemap) and the server's path to the map file. Templeton attempts to resolve the map file's path from the applications path, but may fail if the server does not return error 404 (not found) for invalid document paths.
3. Imagemaps may refer to executables. Since Templeton cannot decompile executables, these imagemaps cannot be retrieved.
4. Some servers support map directories where all files in the directory are processed as map resolution files. These servers do not permit the retrieval of the map document. Consequently, map directory references are not supported by Templeton.
Templeton does not support cookies. A "cookie" is a string of data sent from the web server to the web client. The web client is expected to return the cookie information to the server. It has been rumored that some servers are denying access to documents if cookie information is not returned. To date, we have not found any such server, but if they exist, Templeton will not be able to retrieve these documents. (According to the HTTP 1.0 standard and HTTP 1.1 proposed standard, cookies are optional information and cannot be "demanded".) Templeton may support cookies in a later release.

Remote Application Limitations

The following limitations are due to remote applications. These are generally in the form of CGI scripts, Java code, or embedded JavaScript. (They are refered to as remote applications because the come from a remote server.)

New in version 1.970: CGI applications which return a GIF or JPG file do not necessarily have the extension ".gif" or ".jpg". The file containing the retrieved data is the same as the CGI application's name+parameters. Filters such as "deny *.gif" will not work on them. Netscape and WebExplorer both properly view the file as a graphic.
Prior to 1.970, probable CGI applications where placed in their own directories with the extension ".html" instead of ".gif" or ".jpg".
The filename may be changed in a later release.
CGI applications which take optional query arguments may be mistaken for directories or sent varied requests. CGI applications are specified in 3 ways:
1. http://host/path/cgi
  In this case, Templeton will first test if the URL specifies a directory. This is done by appending a '/'. This may change the CGI application's interpretation of the request. If the request succeeds, then the application may be mistaken for a directory specification.
2. http://host/path/cgi/arg1/...
  In this case, the directory structure is built and the final argument becomes the file name. This may cause problems when more optional arguments are identified. For example:
  http://host/path/cgi/arg1
  http://host/path/cgi/arg1/arg2
  There is no way for a web robot (or browser) to know the file specified is actually a CGI application. Templeton will attempt to first resolve "http://host/path/cgi/arg1/" (note the trailing '/'). If this succeeds, then the URL is assumed to specify a directory. This may also alter the CGI interpretation of the application.
  http://host/path/cgi?arg1...
  In this case, the application is correctly identified as a probable CGI application. Any "/" that exist within the query (after the "?") will be translated into a "_" so that directories will not be built. This may cause conflict in the case:
  http://host/path/cgi?arg1/arg2
  http://host/path/cgi?arg1_arg2
  Where these represent two separate CGI queries. (If the "/" into "_" translation is not performed, then the same problem mentioned in #2 can occur.)
The query may be modified when Templeton looks for a directory. It should be noted that most CGI applications tested so far return error code 200 (OK) when given invalid arguments. This leads to the protocol limitation (listed above) where a server returns 200 instead of 404 (or 4xx) for invalid documents.
Templeton does not retrieve Java code. This feature may be available in later releases.
Templeton does not support JavaScript code. JavaScript in HTML documents are not interpreted and may fail to work in retrieved files. All JavaScript entities found when processing a URL are noted in the log files. This feature may be available in later releases.
Embedded <SCRIPT ...> ... </SCRIPT> code will not be interpreded by Templeton. This may break working embedded code when an explicit URL is used. The presence of scripts are noted in the log files. This will be corrected in a later release. Embedded scripts are logged in the log files.

Network Limitations

The following limitations are due to networking implementations.

Templeton may seem to hang when the nameserver is unable to resolve a hostname. This "hanging" is the nameserver request timing out. This interval is system dependent, but is generally no longer than two minutes. Currently, Templeton is single-threaded and at the mercy of this limitation. This may be corrected in later releases.
gethostname() under OS/2 does not resolve properly if the environment variable HOSTNAME is not set. This function also has problems when operated on a gateway using a private LAN and a PPP/SLIP interface, with hostnames that begin with numbers (such as those provided by Worldnet.att.net), and with machine names containing a '.' character. Currently, the user must set their username in the configuration file. This will be corrected in a future release.
Templeton cannot determine if a mounted drive supports long file names. It is assumed that the drive does support long file names, but this may be a bad assumption. For example, a DOS machine can export a partition to a Linux system running Templeton. The user must set the FATFlag in the configuration file to overcome this problem. The DOS version of Templeton (when released) does not support long file names.

Program Limitations

The following limitations are due to the design and implementation of Templeton. Various developmental factors lead to these limitations. Many of these limitations will be changed in later releases.

Command_html, etc. specified in the configuration file cannot be longer than 255 characters. The expanded '%' commands cannot be longer that 1023 characters. Longer commands will be truncated. Note: scripts which require an ending character sequence will not receive the ending sequence if the command is truncated.
The URL used in Command_html, etc. (resolved from "%u", "%p", or "%r") may contain invalid command-line characters including "|", "&", and "?". The command is initiated in the default shell (csh, command.com, cmd.exe, etc.) and the shell characters may be interpreted. The user must ensure that the defined command string does not process these characters. This may be achieved by using double or single quotes (depending on the operating system.)
Maximum URL length is 255 characters. Longer URLs will be truncated. This may be changed in future releases.
Templeton (and other web robots) assume that the previous password for a realm remains valid. This assumption is incorrect when different documents (usually on different servers) coincidently use the same realm string. Templeton only supports one (1) username:password per realm. This may be changed in later releases.
Templeton performs single-pass processing on all documents. If a link is thought to exist (and placed in the processing queue) but later is unretrievable, all previously retrieved documents will contain links to a nonexisting file.
Changed in version 1.960. Templeton only permits a 6K meta header from the server. Longer meta headers may be interpreted as part of the document. This is currently implemented so servers which do not return meta information will not have the entire (huge) file parsed. (The 6K assumes a maximum of 4K in cookie information and 2K is other meta information.) This will be modified in a future release.
Changed in version 1.960 Because Templeton (and all other web robots) is unable to know the default file of a directory (it differs on each server), Templeton is unable to distinguish between '/' and '/default_index_name'. If the server's default index name matches Templeton's 'Index' name (defined in the configuration file) then the default directory file may not be retrieved. This may be modified in a future release.

Rational:
Suppose there are 2 files: Welcome.html and Index.html. Welcome.html is the server's default file name. Index.html (coincidently) is Templeton's default file name. A single HTML file may refer to "/", "/Welcome.html", and "/Index.html". Assuming that the default directory file is the same as Templeton's would incorrectly store a copy of Welcome.html in the local Index.html file. Retrieving the file twice would resolve this issue.
Unfortunately, this also means a default server file called "Index.html" would be retrieved twice (once for "/" and once for "/Index.html"). Worse: if the default index is never specified by an HTML document, but a different remote file called "Index.html" is found, then the default directory data will be lost.
Since this is a Lose-Lose situation, Templeton will retrieve the file "Index.html" only once. Note, prior to version 1.951, Templeton would retrieve the file twice. After much debate, this option was changed.
How's that for a complicated situation?
Regular HTML documents which contain a '?' may be incorrectly identified as a probable CGI application. (They shouldn't have '?' in the URI, but bad HTML code never ceases to amaze me.) A URI specified as "document.html" and "document.html?garbage" will generate two separate files containing the same data. (Most servers are smart enough to ignore the '?garbage' after a valid HTML document.)

[Main Menu]